Simultaneous Query Pipelines in QPipe

Data warehousing and scientific database applications operate on massive datasets and are characterized by complex queries accessing large portions of the database. Concurrent queries often exhibit high data and computation overlap, e.g., they access the same relations on disk, compute similar aggregates, or share intermediate results. Unfortunately, run-time sharing in modern database engines is limited by the paradigm of invoking an independent set of operator instances per query, potentially missing sharing opportunities if the buffer pool evicts data early. QPipe is a new, operator-centric, relational query engine that can detect and exploit overlap across concurrent queries, at run time [1]. In QPipe, each relational operator is promoted to an independent micro-engine (μEngine) with its own resource management and runtime support. Incoming queries break up into as many tasks (or query packets) as the nodes of the compiled query tree plan, and queue up in front of each μEngine. Since query packets are self-contained requests, QPipe allows external applications to submit custom query packets, bypassing the regular parsing and optimizing phases. Under regular query execution, μEngines work independently and evaluate each query in parallel. Data flow for each query between μEngines occurs through dedicated tuple buffers. μEngines continuously monitor their queue to detect data and work sharing opportunities across different queries. Once such an opportunity is detected, only one query packet remains active, performing the overlapping operation, while the results are simultaneously pipelined to all consuming queries. More details about sharing opportunities and the implementation of simultaneous pipelining can be found elsewhere [1]. This system demonstration exposes the key novel features of QPipe, and also provides an intuitive way of visualizing query execution inside the database engine. Unlike modern commercial engines which are typically demonstrated as “black boxes,” due to tight integration of system components and query processing algorithms, QPipe naturally breaks up query execution stages, allowing for a visually appealing demonstration. We organize the demo storyline into three parts: • Resource utilization and query progress. For a single query, we show which μEngines are working in parallel, the progress of the query, and the tuple flow between relational operators. For multiple concurrent queries, we show queue status and thread assignment at each μEngine, along with opportunities for sharing data or work across queries. • Simultaneously pipelined query execution. By switching QPipe to simultaneous pipelining mode, we demonstrate the entire procedure of sharing overlapping table scans or intermediate result computation across different queries. All possible run-time actions of QPipe are demonstrated through pre-designed scenarios. • Ad-hoc query pipelines. QPipe allows external applications to submit ad-hoc query packets. The user is able to construct and submit custom packets to QPipe through a graphical user interface, and observe how these are being evaluated.